Skip to content

Allow opt-out of just-bash dependencies - #33

Open
aron-cf wants to merge 1 commit into
mainfrom
modularize-just-bash
Open

Allow opt-out of just-bash dependencies#33
aron-cf wants to merge 1 commit into
mainfrom
modularize-just-bash

Conversation

@aron-cf

@aron-cf aron-cf commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

The worker backend embeds the entire just-bash shell in SHELL_MODULES, a single frozen record of moduleName → { js } that WorkerBackend spreads into the Worker Loader callback. A handful of optional commands dominate that upload — curl pulls in undici (~620 KB) and html-to-markdown pulls in domino (~555 KB), with python, sqlite, and js-exec behind them — and every consumer paid for all of them because there was no seam to drop a command.

This splits the bundle into independently-droppable feature groups so a consumer can remove commands they don't need at build time, without any change to the default output.

What changed

build-bundle.mjs now partitions esbuild's emitted modules into a core group plus one group per optional command, and writes each to generated/<group>.ts (gitignored, regenerated on prepare/pretest/pretypecheck). Assignment is by reachability, not by name:

a chunk goes to feature F  ⟺  F is its only reacher AND core can't reach it
otherwise                  →  core   (shared chunks, anything a kept command needs)

so dropping one feature can never break another. undici is attributed to curl explicitly, because just-bash loads it from the shell body rather than through the curl command chunk.

A new checked-in shell-modules.ts reassembles SHELL_MODULES from the groups, importing each through its own published subpath:

import coreModules from "@cloudflare/computer/shell/core";
import curlModules from "@cloudflare/computer/shell/curl";
import htmlToMarkdownModules from "@cloudflare/computer/shell/html-to-markdown";
// … python, sqlite, js-exec

export const SHELL_MODULES = Object.freeze({
  ...coreModules,
  ...curlModules,
  ...htmlToMarkdownModules,
  // …
});

Because the groups are imported by subpath, a consumer can swap any one for an empty module table with a wrangler alias, and esbuild then never pulls that group — or its heavy dependency — into the uploaded Worker.

Consumer config

Drop a feature in wrangler.jsonc:

"alias": {
  "@cloudflare/computer/shell/curl": "@cloudflare/computer/empty",
  "@cloudflare/computer/shell/html-to-markdown": "@cloudflare/computer/empty"
}

Optional features and the rough weight each one removes from the upload when aliased out (generated group source):

Feature Alias subpath ~Size removed (raw) (gzip) Main dependency
curl @cloudflare/computer/shell/curl ~646 KiB ~172 KiB undici (network stack)
html-to-markdown @cloudflare/computer/shell/html-to-markdown ~571 KiB ~100 KiB domino (HTML DOM)
sqlite @cloudflare/computer/shell/sqlite ~104 KiB ~24 KiB sql.js
js-exec @cloudflare/computer/shell/js-exec ~12 KiB ~5 KiB
python @cloudflare/computer/shell/python ~11 KiB ~4 KiB

For reference, core (never removable) is ~2,346 KiB raw / ~538 KiB gzip. Leaving the alias out ships the whole shell, byte-for-byte the previous single-file bundle. A dropped command still parses; invoking it after its chunk is gone fails at runtime with a module-not-found.

The example Worker drops curl and html-to-markdown, which takes its upload from ~5,956 KiB → ~4,774 KiB raw (~1,344 → ~1,071 KiB gzip). Default build (nothing aliased) is unchanged at ~5,956 KiB (538 KiB gzip).

Alternative considered: define constants

A per-feature define constant was considered — one boolean per feature that gates its import:

"define": {
  "__COMPUTER_INCLUDE_CURL__": "false",
  "__COMPUTER_INCLUDE_HTML_TO_MARKDOWN__": "false"
}
const SHELL_MODULES = Object.freeze({
  ...coreModules,
  ...(__COMPUTER_INCLUDE_CURL__ ? curlModules : {}),
  ...(__COMPUTER_INCLUDE_HTML_TO_MARKDOWN__ ? htmlToMarkdownModules : {}),
});

This does drop the bytes (verified against esbuild), since esbuild constant-folds the dead branch and tree-shakes the group away — but only when the generated groups are marked sideEffects: false, and it needs a typeof guard on each constant so features still default on when a consumer sets nothing. It's also still one define entry per feature. alias achieves the same removal with no sideEffects hint and no magic globals, so it's what this PR ships.

Follow-ups

This PR only removes optional command chunks; core (~2,346 KiB raw / ~538 KiB gzip) is unchanged. Dead-code elimination has already run, so what's left is reachable, but a few levers could shrink it further:

  • Minify the generated groups. build-bundle.mjs currently emits the embedded module sources unminified. Adding minify would take ~617 KiB raw / ~66 KiB gzip off core (and ~296/30 off html-to-markdown, ~121/11 off curl) with no functionality lost — the largest single win.
  • Split more always-on commands. awk (a full language interpreter), sed, tar, and git live in core today but are the kind of thing many consumers don't need; they could become optional groups on the same alias mechanism.
  • Swap heavy shared deps. core carries re2js (the RE2 regex engine, ~225 KiB) for RE2-compatible semantics and pako (~128 KiB, via isomorphic-git). Falling back to native RegExp where semantics allow, and to Workers' node:zlib for pako, would remove both.

Testing

  • npm run typecheck — passes.
  • npx biome check . — passes.
  • npx vitest run (package unit tests) — 799 passed; npm run test:worker-backend (real workerd Loader path) — 4 passed.
  • wrangler deploy --dry-run on the example: 5,956 KiB with no alias, 4,774 KiB with curl + html-to-markdown dropped.

Notes

The generated src/backends/worker/generated/ files are gitignored; run npm run build:shell-bundle --workspace @cloudflare/computer after applying to regenerate them before building.

The worker backend embeds the whole just-bash shell in
SHELL_MODULES, and a handful of optional commands dominate the
upload: curl carries undici (~620 KB) and html-to-markdown
carries domino (~555 KB), with python, sqlite, and js-exec behind
them. Every consumer paid for all of them because the modules
shipped as one frozen record with no seam to drop a command.

build-bundle.mjs now partitions the emitted modules into a core
group plus one group per optional command and writes them under
generated/ as separate modules. A chunk lands in a feature group
only when that feature is its sole reacher and core cannot reach
it, so shared chunks and anything a kept command needs stay in
core and dropping one feature never breaks another. undici is
attributed to curl by hand because just-bash loads it from the
shell body rather than through the curl command chunk.

shell-modules.ts assembles SHELL_MODULES from the groups, importing
each through its @cloudflare/computer/shell/<feature> subpath so a
consumer can swap one for @cloudflare/computer/empty with a
wrangler alias. That drops the feature's exclusive chunks, heavy
dependency included, from the uploaded Worker. Leaving the alias
out ships the whole shell, byte-for-byte the single-file bundle.
The command still parses; invoking it after its chunk is gone
fails at runtime with a module-not-found.

The worker example drops curl and html-to-markdown, taking its
upload from ~5,956 KiB to ~4,774 KiB raw. The package and worker
docs describe the mechanism and the available features.
@pkg-pr-new

pkg-pr-new Bot commented Jul 31, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/cloudflare/computer/@cloudflare/computer@33

commit: 4dbf7d8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant